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Abstract 

This paper presents a computation of the V~, dimension for regression in bounded subspaces 
of Reproducing Kernel Hilbert Spaces (RKHS) for the Support Vector Machine (SVM) 
regression e-insensitive loss function, and general L p loss functions. Finiteness of the V^ 
dimension is shown, which also proves uniform convergence in probability for regression 
machines in RKHS subspaces that use the L e or general L p loss functions. This paper 
presents a novel proof of this result also for the case that a bias is added to the functions 
in the RKHS. 
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1 Introduction 

In this paper we present a computation of the V 1 dimension of real- valued functions L(y, /(x)) = 
\y — /(x)| p and (Vapnik's e-insensitive loss function [10]) L(y,f(x)) — \y — /(x)| e with / in a 
bounded sphere in a Reproducing Kernel Hilbert Space (RKHS). We show that the V^ dimension 
is finite for these loss functions, and compute an upper bound on it. We present two solutions to 
the problem. First we discuss a simple argument which leads to a loose upper bound on the V^ 
dimension. Then we refine the result in the case of infinite dimensional RKHS, which is often the 
type of hypothesis spaces considered in the literature (i.e. Radial Basis Functions [8, 5]). Our 
result applies to standard regression learning machines such as Regularization Networks (RN) 
and Support Vector Machines (SVM). We also present a novel computation of the V 1 dimension 
when a bias is added to the functions, that is with / being of the form f — fo + b, where b G R 
and /o is in a sphere in an infinite dimensional RKHS. 

For a regression learning problem using L as a loss function it is known [1] that finiteness of 
the V 1 dimension for all 7 > is a necessary and sufficient condition for uniform convergence in 
probability [10] . So the results of this paper have implications for uniform convergence both for 
RN and for SVM regression [4]. 

Previous related work addressed the problem of pattern recognition where L is an indicator 
function [?, 6]. The fat-shattering dimension [1] was considered instead of the V 1 one. A different 
approach to proving uniform convergence for RN and SVM is given in [12] where covering number 
arguments using entropy numbers of operators are presented. In both cases, regression as well 
as the case of non-zero bias b were marginally considered. 

The paper is organized as follows. Section 2 outlines the background and motivation of this work. 
The reader familiar with statistical learning theory and RKHS can skip this section. Section 3 
presents a simple proof of the results as well as some upper bounds to the V^ dimension. Section 
4 presents a refined computation in the case of infinite dimensional RKHS, as well as the case 
where the hypothesis space consists of functions of the form f — fo + b where b e R and fo in a 
sphere in a RKHS. Finally, section 5 discusses possible extensions of this work. 

2 Background and Motivation 

We consider the problem of learning from examples as it is viewed in the framework of statistical 
learning theory [10]. We are given a set of / examples {(xi, yi), .., (x/, yi)} generated by randomly 
sampling from a space X x Y with X C R d , Y C R according to an unknown probability 
distribution P(x,y). Throughout the paper we assume that X and Y are bounded. Using this 
set of examples the problem of learning consists of finding a function / : X — > Y that can be 
used given any new point xelto predict the corresponding value y. 

The problem of learning from examples is known to be ill-posed [10, 9]. A classical way to solve it 
is to perform Empirical Risk Minimization (ERM) with respect to a certain loss function, while 
restricting the solution to the problem to be in a "small" hypothesis space [10]. Formally this 
means minimizing the empirical risk J emp [/] = \ YJ i=1 L{y^ /(x,)) with / e H, where L is the 
loss function measuring the error when we predict /(x) while the actual value is y, and 7i is a 
given hypothesis space. 
In this paper, we consider hypothesis spaces of functions which are hyperplanes in some feature 



space: 

oo 
/( X ) = ^W n 0„(x) (1) 

n=l 

with: 

oo 2 
n=l A ™ 

where ra (x) is a set of given, linearly independent basis functions, X n are given non-negative 
constants such that Y^Li ^n < °°- Spaces of functions of the form (1) can also be seen as 
Reproducing Kernel Hilbert Spaces (RKHS) [2, 11] with kernel K given by: 

oo 

^( x , y) = ]C A„0 ra (x)0 ra (y). (3) 

n=l 

For any function / as in (1), quantity (2) is called the RKHS norm of /, ||/||#, while the number 
D of features <f) n (which can be finite, in which case all sums above are finite) is the dimensionality 
of the RKHS. 

If we restrict the hypothesis space to consist of functions in a RKHS with norm less than a 
constant A, the general setting of learning discussed above becomes: 

Minimize: \ £{ =1 L( Vi , /(x,)) 
subject to: \\f\\% < A\ (4) 

An important question for any learning machine of the type (4) is whether it is consistent: as 
the number of examples (xj, yi) goes to infinity the expected error of the solution of the machine 
should converge in probability to the minimum expected error in the hypothesis space [10, 3]. In 
the case of learning machines performing ERM in a hypothesis space (4), consistency is shown to 
be related with uniform convergence in probability [10], and necessary and sufficient conditions 
for uniform convergence are given in terms of the V 1 dimension of the hypothesis space considered 
[1, 7], which is a measure of complexity of the space. 

In statistical learning theory typically the measure of complexity used is the VC-dimension. 
However, as we show below, the VC-dimension in the above learning setting in the case of 
infinite dimensional RKHS is infinite both for L, p and L e , so it cannot be used to study learning 
machines of the form (4). Instead one needs to consider other measures of complexity, such as 
the Vy dimension, in order to prove uniform convergence in infinite dimensional RKHS. We now 
present some background on the V 1 dimension [1]. 
In the case of indicator functions the definition of the V 1 dimension is: 

Definition 2.1 The Vy- dimension of a set of indicator functions {#(/(x)), / G Ti] (where 9 is 
the heavy side function), is the maximum number h of vectors x 1; . . . , x ft that can be separated 
into two classes in all 2 h possible ways using functions of the set and the rules: 

class 1 if: f (x) > s + 7 
class -1 if: /(x) < s — 7 

for some s > 0. //, for any number N, it is possible to find N points xi, . . . ,xat that can be 
separated in all the 2 N possible ways, we will say that the V^-dimension of the set is infinite. 



For 7 = s = this becomes the definition of the VC dimension of a set of indicator functions 
0(/(x)) [10]. Moreover, in the case of hyperplanes (1), the V^ dimension has also been referred 
to in the literature [10] as the VC dimension of hyperplanes with margin. In order to avoid 
confusion with names, we call the VC dimension of hyperplanes with margin as the V^ dimension 
of hyperplanes (for appropriate 7 depending on the margin, as discussed below). The VC 
dimension has been used to bound the growth function Q n {l). This function measures the 
maximum number of ways we can separate I points using functions from hypothesis space 7i. If 
h is the VC dimension, then Q n (l) is 2 l if / < h, and < (^ ) h otherwise [10]. 
In the case of real valued function (which is the case we are interested in here) the definition of 
the V.y dimension is: 

Definition 2.2 Let C < L(y,f(x)) < B, f e H, with C and B < 00. The V 7 - dimension 
of L in Ti (of the set {L(y, f(x)), f e 7i}) is defined as the maximum number h of vectors 
(xi,?/i) . . . , (-Xh,Vh) that can be separated into two classes in all 2 h possible ways using rules: 

class 1 if: L(y h f{xi)) > s + 7 
class -1 if: L(y u f{xi)) < s - 7 

for f e 7i and some C + ^<s<B — 7. //, for any number N , it is possible to find N 
points (xi, 2/1) . . . , (xjy, 2/jv) that can be separated in all the 2 N possible ways, we will say that the 
V^-dimension of L in 71 is infinite. 

For 7 = and for s being free to change values for each separation of the data, this becomes the 
VC dimension of the set of functions [10]. 

Using the V 1 dimension Alon et al. [1] gave necessary and sufficient conditions for uniform 
convergence in probability to take place in a hypothesis space 7i. In particular they proved the 
following important theorem: 

Theorem 2.1 (Alon et al. , 1997 ) Let C < L(y,f(x))) < B, f e H, H be a set of bounded 
functions. The ERM method uniformly converges (in probability) if and only if the V n dimension 
of ' L in 71 is finite for every 7 > 0. 

It is clear that if for learning machines of the form (4) the V^ dimension of the loss function L 
in the hypothesis space defined is finite for V7 > 0, then for these machines uniform convergence 
takes place. In the next section we present a simple proof of the finiteness of the V^ dimension, 
as well as an upper bound on it. 

2.1 Why not use the VC-dimension 

Consider first the case of L p loss functions. Consider an infinite dimensional RKHS, and the set 
of functions with norm \\f\\\ < A 2 . If for any iV we can find N points that we can shatter using 
functions of our set according to the rule: 

class 1 if : \y — /(x)| p > s 
class — 1 if : \y — /(x)| p < s 

then clearly the VC dimension is infinite. Consider N points (xj,|/j) with yi = for all i, and 
Xj be such that the smallest eigenvalue of matrix G with Gij = i^(xj,Xj) is > A. Since we are 



in infinite dimensional RKHS, matrix G is always invertible [11], so A > since G is positive 

definite. 

For any separation of the points, we consider a function / of the form /(x) = J2i=i CKj-K^x^x), 

which is a function of the form (1). We need to show that we can find coefficients ol{ such that 

the RKHS norm of the function is < A 2 . Notice that the norm of a function of this form is ol t Gol 

where (ot)i = a» (throughout the paper bold letters are used for noting vectors). Consider the 

set of linear equations 

N - 

Xj G class 1 : Z)*=i cxiGij = sp + rj r] > 

N - 

Xj G class — 1 : Z)i=i a iGij = s p — r\ rj > 

Let s = 0. If we can find a solution ot, to this system of equations such that ol t Gol < A 2 we can 
perform this separation, and since this is any separation we can shatter the N points. Notice 
that the solution to the system of equations is G~ l r] where t] is the vector whose components 
are (rj)i = rj when Xj is in class 1, and — rj otherwise. So we need {G~ l rfj r G{G~ 1 ri) < A 2 => 
t} T G lr rj < A 2 . Since the smallest eigenvalue of G is > A > 0, rj T G ' lr rj < ' x ' . Moreover 
r} T t} = Nrf. So if we choose r\ small enough such that —?- < A 2 =^ rf < ^^, the norm of the 
solution is less than A 2 , which completes the proof. 

For the case of the L e loss function the argument above can be repeated with y^ — e to prove 
again that the VC dimension is infinite in an infinite dimensional RKHS. 

Finally, notice that the same proof can be repeated for finite dimensional RKHS to show that 
the VC dimension is never less than the dimensionality D of the RKHS, since it is possible to 
find D points for which matrix G is invertible and repeat the proof above. As a consequence the 
VC dimension cannot be controlled by A 2 . 

3 A simple upper bound on the V 1 dimension 

Below we always assume that data X are within a sphere of radius R in the feature space defined 
by the kernel K of the RKHS. Without loss of generality, we also assume that y is bounded 
between —1 and 1. Under these assumptions the following theorem holds: 

Theorem 3.1 The V^ dimension h for regression using L p or L e loss functions for hypothesis 
spaces 7i A = {/(x) = Y£?=i w n ( l ) n(' x -) I Y^Li x°" — A 2 } and y bounded, is finite for V7 > 0. If D 
is the dimensionality of the RKHS, then h < 0(min(D, - — ^{ — ))■ 

Proof. Let's consider first the case of the L\ loss function. Let B be the upper bound on the 
loss function. From definition 2.2 we can decompose the rules for separating points as follows: 

class 1 if yi- /(xj) > s + 7 

or /(xj) - y { > s + 7 _ 

class - 1 if yi- /(x^ < s - 7 

or /(x<) - yi < s - 7 

for some 7 < s < B — 7. For any N points, the number of separations we can get using rules 
(5) is not more than the number of separations we can get using the product of two indicator 



functions (of hyperplanes with margin): 

function (a) : class 1 if y, — /i(xj) > Si + 7 

class - 1 if i/i- /i(xj) < si -7 ,. 

function (b) : class 1 if / 2 (xj) — Hi > £2 + 7 

class - 1 if / 2 (xi) - i/j < s 2 - 7 

where /1 and / 2 are in 7i A , 7 < Si, s 2 < S — 7. For Si = s 2 = s and for /1 = f 2 = f we recover 
(5): for example, if y — /(x) > s + 7 then indicator function (a) will give — 1, indicator function 
(b) will give also — 1, so their product will give +1 which is what we get if we follow (5). So since 
we give more freedom to /1, / 2 , Si, s 2 clearly we can get at least as many separations for any set 
of points than the number of separations we would get using (5). 

As mentioned in the previous section, for any N points the number of separations is bounded 
by the growth function. Moreover, for products of indicator functions it is known [10] that the 
growth function is bounded by the product of the growth functions of the indicator functions. 
Furthermore, the indicator functions in (6) are hyperplanes with margin in the D + l dimensional 
space of vectors {0 n (x),y} where the radius of the data is R 2 + 1, the norm of the hyperplane 
is bounded by A 2 + 1, (where in both cases we add 1 because of y), and the margin is at 

2 

least jj +1 - The Vy dimension /i 7 of these hyperplanes is known [10, ?] to be bounded by /i 7 < 

min((Z} + 1) + 1, - — ^—^ — )■ So the growth function of the separating rules (5) is bounded by 

Q(l) < {^) hl {'^) h " f whenever I > h 1 . If h r ^ 9 is the Vy dimension, then K'^ g cannot be larger 

than the larger number I for which inequality 2 l < (f^)* 1 " 1 (T t L ) h ~ l holds. From this we get that 

/ < 5/i 7 , therefore h^ eg < 5 min (D + 2, ^ -) which proves the theorem for the case of L\ 

loss functions. 

For general L p loss functions we can follow the same proof where (5) now needs to be rewritten 

as: 

class 1 if Hi- /(xj) > (s + 7)? 

or f(x i )-y i >(s + 'y)f , ? x 

class - 1 if yi- /(xj) < (s - 7)? 
or /(xj) -yi < (3-7)' 

Moreover, for p > 1, (s + 7)? > sp + -^ (since 7 = ((s + j)*) — [sp ) < ((s + 7)^ — sp)(pB)) 
and (s — 7)5 < sp — -^ (similarly). Repeating the same argument as above, we get that the V^ 

dimension is bounded by 5 min (D + 2, (p - - — 2^ -). Finally, for the L e loss function (5) can 

be rewritten as: 

class 1 if yi — /(xj) > s + 7 + e 

or f(x i )-y i >s + 'y + e , g . 

class — 1 if ?/j — /(xj) < s — 7 + e 
or /(xj) -2/i<s-7 + e 

where calling s' = s + e we can simply repeat the proof above and get the same upper bound on 
the V^ dimension as in the case of the L\ loss function. (Notice that the constraint 7 < s < B — 7 
is not taken into account. We believe that taking this into account may slightly change the V^ 
dimension for L e ). □ 

Notice that these results imply that in the case of infinite dimensional RKHS the V^ dimension is 
still finite and is influenced only by 5^ — ^—^ — — . In the next section we present a more refined 
upped bound of the V^ dimension in the infinite dimensional case. 



4 A more refined computation of the V 1 dimension 

Below we assume that the data x are bounded so that for any TV x TV matrix G with entries 
Gij = K(xi,Xj) (where K is, as mentioned in the previous section, the kernel of the RKHS 
considered) the largest eigenvalue of G is always < R 2 . Notice that R 2 is related to the radius 
of the data x in the "feature space" {0 n (x)} typically considered in the literature [10, ?, 6]. We 
also note with B the upper bound of L(y, /(x)). 



Theorem 4.1 The Vy dimension for regression using L\ loss function and for hypothesis space 

oo w{ 
n=l X n 

B?A 2 ~ 



H A = {f(x) = En=i WnM*) + b | E^°=i ^<A 2 } is finite for V7 > 0. In particular: 



1. Ifbis constrained to be zero, then V y < 

2. If b is a free parameter, V y < 4 



7 

R 2 A 2 



Proof of part 1. 

Suppose we can find TV > ^-§- points {(xi,yi), ..., (xn,Vn)} that we can shatter. Let s G 
[7, B — 7] be the value of the parameter used to shatter the points. 

Consider the following "separation" 1 : if \yi\ < s, then (xi,yi) belongs in class 1. All other points 
belong in class -1. For this separation we need: 

\yi-f(xi)\ > s + 7, if \Vi\ < s , Q . 

\yi- f(xi)\ < 5-7, if \vi\ >s 

This means that: for points in class 1 / takes values either yi + s + 7 + Si or yi — s — 7 — Si, for 
Si > 0. For points in the second class / takes values either yi + s — 7 — Si or yi — s + 7 + Si, for 
Si G [0, (s — 7)]. So (9) can be seen as a system of linear equations: 

00 
^w ra n (xi) = U. (10) 

n=l 

with U being y t + s + 7 + S u or y { - s - 7 - S u or ^ + s - 7 - Si, or ^ - s + 7 + S u depending 
on i. We first use lemma 1 to show that for any solution (so i, are fixed now) there is another 
solution with not larger norm that is of the form X^Li djiC(xj,x). 

Lemma 4.1 Among all the solutions of a system of equations (10) the solution with the minimum 
RKHS norm is of the form: Yjf=\ onKiy^i, x) with ex. = G~ l t. 

For a proof see the Appendix. Given this lemma, we consider only functions of the form 
Yjf=\ otiK(xi,x). We show that the function of this form that solves the system of equations 
(10) has norm larger than A 2 . Therefore any other solution has norm larger than A 2 which 
implies we cannot shatter TV points using functions of our hypothesis space. 
The solution a = G~H needs to satisfy the constraint: 

a T G<x = t T G-H < A 2 



1 Notice that this separation might be a "trivial" one in the sense that we may want all the points to be +1 or 
all to be -1 i.e. when all \yi\ < s or when all \yi\ > s respectively. 

6 



Let \ max be the largest eigenvalue of matrix G. Then t T G l t > -r—^-. Since \ max < R 2 , 

t T G~ 1 t > -^2-. Moreover, because of the choice of the separation, t T t > JV7 2 (for example, for the 
points in class 1 which contribute to t T t an amount equal to (|/i + s+7+5j) 2 : |j/j| < s =$> Di + s > 0, 
and since 7+5j > 7 > 0, then (|/j+s+7+5j) 2 > 7 2 . Similarly each of the other points "contribute" 
to t T t at least 7 2 , so t T t > /V7 2 ). So: 

eo-H >^->a 2 

R 1 

since we assumed that TV > R 2 . This is a contradiction, so we conclude that we cannot get 
this particular separation. □ 

Proof of part 2. 

Consider N points that can be shattered. This means that for any separation, for points in the 
first class there are Si > such that \f(xi) + b — yi\ = s + 7 + Si. For points in the second class 
there are Si G [0, s — 7] such that \f(xi) +b — yi\ = s — 7 — Si. As in the case b = we can remove 
the absolute values by considering for each class two types of points (we call them type 1 and 
type 2). For class 1, type 1 are points for which f(xi) = yi + s + 7 + Si — b = ti — b. Type 2 are 
points for which f(xi) = Hi — s — 7 — Si — b = ti — b. For class 2, type 1 are points for which 
f(x{) = yi + s — 7 — Si — b = ti — b. Type 2 are points for which f(x{) = yi — s + 7 + Si — b = ti — b. 
Variables U are as in the case 6 = 0. Let Sn, Sn, S'-n, >5-i2 denote the four sets of points (SV/ 
are points of class % type j). Using lemma 1, we only need to consider functions of the form 
f(x) = 12iLi (XiK(xi,x). The coefficients «« are given by a. = G~ l (t — b) there 6 is a vector of 
Vs. As in the case 6 = 0, the RKHS norm of this function is at least 

(11) 

The 6 that minimizes (11) is jj(52iLiU)- So (11) is at least as large as (after replacing 6 and 
doing some simple calculations) 
We now consider a particular separation. Without loss of generality assume that y\ < y^ < 
■ ■ ■ < Vn an d that N is even (if odd, consider N — 1 points). Consider the separation where 
class 1 consists only of the "even" points {N, N — 2, ... , 2}. The following lemma is shown in 
the appendix: 

Lemma 4.2 For the separation considered, X)fj=i(^ — tj) 2 ^ s °>t least as large as 7 - 2 ~ - . 
Using Lemma 4.2 we get that the norm of the solution for the considered separation is at least 





R 2 


(t~ 


bf(t - 


b). 


(El 
1 

2NR 2 


iU). 


So 
=i(*. 


(11) is 

-hf. 


at 1 



R 2 A 2 



as large as 7 ^^2 ■ Since this has to be < A 2 we get that N — -^ < 4 

the proof (assume N > 4 and ignore additive constants less than 1 for simp 

□ 



, which completes 
icity of notation). 



In the case of L p loss functions, using the same argument as in the previous section we get 
that the V^ dimension in infinite dimensional RKHS is bounded by (p ' 2 in the first case of 

theorem 4.1, and by A p ' 2 in the second case of theorem 4.1. Finally for L e loss functions 

the bound on the V^ dimension is the same as that for L\ loss function, again using the argument 
of the previous section. 



5 Conclusion 

We presented a novel approach for computing the V^ dimension of RKHS for L p and L e loss 
functions. We conclude with a few remarks. First notice that in the computations we did not 
take into account e in the case of L e loss function. Taking e into account may lead to better 
bounds. For example, considering \f(x) — y\^,p > 1 as the loss function, it is clear from the proofs 

presented that the V^ dimension is bounded by p - ~ e ' 2 . However the influence of e seems to 

be minor (given that e << B). Furthermore, it may be possible to extend the computations for 
more general loss functions. 

An interesting observations is that the eigenvalues of the matrix G appear in the computation 
of the V 1 dimension. In the proofs we took into account only the largest and smallest eigen- 
values. If similar computations are made to compute the number of separations for a given set 
of points, then it is possible that all the eigenvalues of G are taken into account. This leads 
to interesting relations with the work in [12]. Finally, the bounds on the V^ dimension can be 
used to get bounds on the generalization performance of regression machines of the form (4) [1,4]. 
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Appendix 

Proof of Lemma 4.1 

We introduce the N x oo matrix A in = y / X^0 n (x i ) and the new variable z n = -j=. We can write 
system (10) as follows: 

Az = t. (12) 

Notice that the solution of the system of equation 10 with minimum RKHS norm, is equivalent 
to the Least Square (LS) solution of equation 12. Let us denote with z° the LS solution of system 
12. We have: 

z° = (A T A) + A T t (13) 

where + denotes pseudoinverse. To see how this solution looks like we use Singular Value 
Decomposition techniques: 

A = UEV T , 
A T = VYXJ T , 

from which A T A = VY?V T and (A T A) + = V N T,~ 2 V^ , where E^ 1 denotes the N xN matrix whose 
elements are the inverse of the nonzero eigenvalues. After some computations equation (13) can 
be written as: 

z° = VE-'Ult = {VX N UZ){U N Z- 2 U];)t = AG-H. (14) 

Using the definition of z° we have that 



^2 u>°0„(x) = J2 J2 V A„0 n (x)A ni a i . (15) 



n=l n=l i=\ 

Finally, using the definition of A in we get: 



^w°0„(x) = ^2K(x,Xi)ai 

n=l i=l 

which completes the proof. □ 

Proof of Lemma 4.2 

Consider a point {xi,yi) in 5ii and a point (xj,i/j) in S-u such that yi > yj (if such a pair 
does not exist we can consider another pair from the cases listed below). For these points 
(ti - tj) 2 = (y i + s + -f + S i -y j -s + -f + 5jf = ((^ - %•) + 27 + S t + 5j) 2 > 4j 2 . In a similar 



way (taking into account the constraints on the S^s and on s) the inequality (tj — tj) 2 > 4 / -/ 2 can 
be shown to hold in the following two cases: 



(xi, y t ) G Sn, (xj, yj) e S-u U S-12, Vi > Vj 
fa, yi) e S 12 , (xj, yj) e S-u U 5_i2, y t < yj 



(16) 



Moreover, clearly 



A? 



»J=1 



E 



E (**-*i) 2 | + E| E 

\js5_ii|j5_i2,j/i>% / *e5i2 ys5_ii|j5_i2,j/i<% 



(<i - tj)' 



Using the fact that for the cases considered (tj — tj) 2 > 47 2 , the right side is at least 

87 2 J2ieSn (number of points j in class — 1 with yi > Dj) + 
+87 2 Sie5i2 (number of points j in class — 1 with yi < yj) 



(17) 



Let I\ and I 2 be the cardinalities of Sn and Si 2 respectively. Because of the choice of the 
separation it is clear that (17) is at least 

8 7 2 ((1 + 2 + . . . + h)) + (1 + 2 + . . . + (I 2 - 1))) 

(for example if 7i = 2 in the worst case points 2 and 4 are in Sn in which case the first part of 



(17) is exactly 1+2). Finally, since I\+ 1 2 = y, (17) is at least 87 
the lemma. □ 



„2 iV 2 -4 7^(^-4) 
16 2 ' 



which proves 
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